Accelerate Utilities Follow-up #224
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
dsikka
pushed a commit
that referenced
this pull request
Dec 20, 2024
* wip * add modify_offload_module * update docs * WIP * cleanup functions, begin depreciation Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * remove extra space Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * revert get_offloaded_device Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * update to align_module_device Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * add requires skip for accelerate * fix per token initialization * remove align_module_device * respond to nits Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * Accelerate Utilities Follow-up (#224) * rename * implement recursive case * remove print * support OffloadedWeightsLoader * add lifecycle docstring * implement offload_to_weights_map with recursive definition Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * add docstring Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * fix type hint * add check_accelerate guard Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * make device used by clearer Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * update update_prefix_dict Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * reuse fixture Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * use apply rather than recursion Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * clearer delete_from_weights_map * add offload_device argument (#228) Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> --------- Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Etelis
added a commit
to Etelis/compressed-tensors
that referenced
this pull request
Sep 11, 2025
* wip * add modify_offload_module * update docs * WIP * cleanup functions, begin depreciation Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * remove extra space Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * revert get_offloaded_device Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * update to align_module_device Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * add requires skip for accelerate * fix per token initialization * remove align_module_device * respond to nits Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * Accelerate Utilities Follow-up (vllm-project#224) * rename * implement recursive case * remove print * support OffloadedWeightsLoader * add lifecycle docstring * implement offload_to_weights_map with recursive definition Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * add docstring Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * fix type hint * add check_accelerate guard Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * make device used by clearer Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * update update_prefix_dict Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * reuse fixture Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * use apply rather than recursion Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> * clearer delete_from_weights_map * add offload_device argument (vllm-project#228) Signed-off-by: Kyle Sayers <kylesayrs@gmail.com> --------- Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
update_offload_dataand expand capabilitiesChanges
align_module_device, which has already been upstreamed and is just copied here until we bump the requiredaccelerateversiondisable_hf_hookand utilize ininitialize_module_for_quantizationregister_offload_parameterupdate_offload_datawhere data would not be updated properly if the module was offloaded and weights_map dataset was being improperly gottenupdate_offload_dataTesting
tests/test_utils/test_offload.pytests/test_quantization/lifecycle/test_initialize.py